Sentence-based Document Size Reduction

In this article we present a novel document size reduction method that selects characteristic sentences by recognising fundamental semantical structures. With the help of document size reduction, document clustering processes less information, while also avoids misleading content. Sentence selection is carried out in two steps. First, a graph representing fundamental sentence relationships, measured by the number of common words, is constructed. Second, various statistical properties of this graph is computed and fed to a backpropagated neural network, which then chooses a small fraction of sentences deemed to be relevant. Preliminary experiments employing the Reuters-21578 news corpus proved that selection of lead sentences (which summarise each news article) can be more reliably performed based on the sentence relationship graph than on the traditional tf and tf×idf measurements. Experiments showed that the presented method can substitute tf and tf×idf for document clustering.